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INTRODUCTION: 


The  subject  of  this  investigation  pertains  to  usability  and  automated  clinical  practice 
guidelines  (aCPGs)  in  AHLTA,  the  military  health  system’s  (MHS)  electronic  health 
record  (EHR).  The  purpose  of  the  study  is  to  examine  critical  usability  issues  in 
AHLTA,  redesign  and  test  these  functions,  and  design  and  evaluate  aCPGs  by 
incorporating  the  improvements  in  usability.  Four  aims  guide  the  work:  (1)  Detennine 
critical  AHLTA  usability  issues  and  develop  a  new  AHLTA  prototype  by  redesigning 
these  AHLTA  functions  based  upon  user-centered  design  principles;  (2)  In  a  laboratory 
setting,  compare  clinicians’  performance  using  the  current  version  of  AHLTA  and  the 
redesigned  functions  in  an  AHLTA  prototype  derived  from  aim  1;  (3)  Using  the  existing 
Veteran’s  Administration  and  Department  of  Defense  (VA/DoD)  asthma  clinical  practice 
guideline  as  a  foundation,  convert  it  to  an  aCPG  in  AHLTA  using  user-centered  design 
principles;  and  (4)  In  a  laboratory  setting,  compare  clinicians’  performance  using  the 
current  asthma  aCPG  alternate  input  mode  (AIM)  form  in  AHLTA  with  the  newly 
designed  aCPG  in  an  AHLTA  prototype  derived  from  aims  1  and  3. 

BODY: 

Task  1:  Complete  preparatory  work  prior  to  the  grant  initiation 

a.  Tripler  Army  Medical  Center  (TAMC)  Scientific  Review  Committee  (SRC): 
approved  the  protocol  with  modifications  on  7  November  2006. 

b.  Institutional  Review  Board  (IRB)  approvals 

1)  TAMC 

a)  Initial  approval  was  granted  on  18  December  2006  with 
stipulations. 

b)  The  start  letter  was  received  on  13  March  2007  following 
completion  of  second  level  review. 

c)  Approval  of  the  first  modification  specifying  asthma  as  the  aCPG 
to  be  studied,  naturalistic  observation,  transcribing  all  qualitative 
data,  and  a  new  onsite  PI  was  granted  on  29  May  2007 

d)  Approval  of  a  second  modification  to  conduct  in-depth  interviews 
to  guide  the  development  of  the  asthma  aCPG  and  recruit  clinician 
volunteers  from  existing  administrative  meetings  to  validate  the 
usability  findings  instead  of  convening  focus  groups  was  approved 
on  23  July  2007. 

e)  Approval  of  a  third  modification  to  allow  access  to  actual  patient 
records  in  AHLTA  during  the  interviews  related  to  aCPG 
development  was  granted  on  26  September  2007. 

f)  The  multiple  modifications,  within  the  six  months  following 
receipt  of  the  start  letter,  reflect  the  considerable  challenge  of 
attempting  to  conduct  a  research  study  that  is  so  operationally 
focused.  These  challenges  will  continue,  especially  as  new 
versions  of  AHLTA  are  released  at  the  centrally  managed 
enterprise  level  from  the  Clinical  Integration  Technology  Program 
Office  (CITPO)  without  consideration  for  existing  issues  and/or 
studies  at  local  (TAMC  and  other)  levels.  For  example,  the  PI  was 
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able  to  arrange  a  virtual  demonstration  of  a  release  that  is  in 
development  and  scheduled  for  deployment  (AHLTA  3.5). 
Information  gathered  in  this  session  on  23  August  2007  raised 
many  questions  in  the  minds  of  the  study  team  regarding  the 
ultimate  scientific  and  operational  contributions  of  the  project 
because  the  protocol  and  proposal  are  based  on  the  current 
AHLTA. 

2)  University  of  Utah — initial  approval  was  granted  on  29  Mar  07 

3)  Second  level  review — rather  than  MRMC,  the  investigators  were 
directed  to  use  TriCare  Management  Activity  (TMA)  for  the  second 
level  review.  TMA  approved  the  study  on  15  FEB  07. 

c.  Pacific  Science  &  Engineering  (PSE)  was  awarded  a  subcontract  on  10 
December  2006.  A  discussion  of  the  initial  work  plan  was  held  involving  the 
PI,  coPIs,  and  PSE  staff. 

d.  TAMC  IMD  Proprietary  Evaluation  Group  (PEG)  approved  the  CRADA. 
Signature  was  obtained  in  January  2007. 

e.  Space  was  allocated  for  the  research  team  in  D  wing  of  TAMC. 

f.  Source  code  for  AHLTA  computer  training  system  (CTS)  was  obtained  from 
CITPO  on  26  January  2007.  This  source  code  was  requested  by  Pacific 
Science  and  Engineering  (PSE),  a  subcontractor,  to  support  and  expedite  the 
development  of  the  revised  AHLTA  prototype  interface.  However,  the  CTS 
could  not  be  used  for  protocol  analysis  because  the  patients  in  the  CTS  were 
not  sufficiently  complex  nor  was  the  CTS  maintained  at  a  level  currently 
representative  of  the  production  environment.  In  addition,  discussions  with 
CITPO  staff  and  the  CTS  developers  confirmed  that  the  CTS  can  not  be 
modified  and  that  updates  to  CTS  are  only  completed  with  major  AHLTA 
releases. 

g.  Morae  software  (used  to  capture  keystrokes)  was  thoroughly  tested  by  PSE. 
Many  issues  were  identified  including  that  Morae  software  is  not  compatible 
with  AHLTA  and  therefore  cannot  be  used  as  planned  in  this  study.  PSE  staff 
obtained  a  beta  version  of  Morae  that  was  compatible  with  AHLTA,  but  it 
adversely  impacted  network  performance.  Consequently,  PSE  developed 
data  collection  software  to  capture  audio,  video  and  observer  notes  that  were 
time-locked  to  one  another. 

h.  Statement  of  Work  approved  by  Clinical  Investigation  Regulatory  Office  7 
March  2007. 

Task  2:  Complete  grant  start-up 

a.  Project  director  hiring 

1)  The  position  was  filled  on  4  December  2006  and  vacated  on  13  April  2007 
due  to  an  unanticipated  family  issue. 

2)  A  second  project  director  was  hired  on  1  Apr  2006.  She  submitted  a  letter 
of  resignation  effective  29  June  2007.  The  job  description  was  recrafted 
by  the  PI  and  co-PIs  based  on  feedback  from  the  second  project  director 
and  in  consideration  of  the  needs  of  the  study.  Rather  than  a  project 
director,  a  project  manager  was  recruited. 
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3)  A  project  manager  was  hired  with  a  start  date  of  27  August  2007.  A  co-PI 
made  a  site  visit  5-8  SEP  07  to  orient  the  project  manager. 

b.  Project  plan 

1)  A  high-level  project  plan  was  developed  during  the  second  quarter 

2)  The  project  plan  is  under  revision  to  adjust  for  the  various  modifications  to 
the  protocol  as  well  as  the  need  to  de-identify  June  data  prior  to 
proceeding  with  further  data  collection 

c.  Kick-off  meeting  held  at  TAMC  18  June  2007 

1)  Participants  included  the  PI,  co-PIs,  local  PI,  project  director  and  staff 
from  PSE 

2)  Topics  covered  were: 

a)  Project  history 

b)  TAMC  culture  and  complexity 

c)  Relationships  among  TATRC,  the  Hui,  and  The  Geneva  Foundation 

d)  Discussion  of  AMEDD  Centers  of  Excellence  including  the  current 
status  of  the  concept  and  their  implementation 

e)  Impact  of  Defense  Business  Transformation  policies  on  the  project 

f)  Project  priorities  for  this  site  visit 

3)  Additionally,  the  PI,  coPIs  and  lead  staff  member  from  PSE  met  to  review 
the  study  methodology  and  earmark  potential  dates  for  future  work  onsite 
(e.g.,  aCPG  interviews  in  September,  piloting  the  AHLTA  prototype  in 
December) 

d.  Advertise  the  study 

1)  The  physician  consultant,  in  collaboration  with  the  TAMC  Deputy 
Commander  for  Clinical  Services  (DCCS),  distributed  an  email  explaining 
the  study  and  encouraging  clinician  participation 

2)  A  study  brochure  was  designed,  printed,  and  distributed  at  key  meetings 
attended  by  TAMC  providers 

3)  The  physician  consultant  and  project  director  attended  departmental 
administrative  meetings  to  explain  the  study 

e.  Recruit  participants  for  the  usability  data  collection  in  June 

1)  The  project  director  recruited  7  clinicians  for  the  usability  interviews 

2)  The  PI  recruited  an  additional  6  clinicians  for  the  usability  interviews 

3)  The  PI  also  recruited  6  clinicians  to  participate  in  the  naturalistic 
observations 

Task  3:  Construct  initial  Functional  Decomposition  Diagrams  (FDDs) 

a.  A  literature  review  was  completed  but  no  FDDs  were  located 

b.  The  FDDs  are  being  developed  by  PSE  from  the  June  data 

Task  4:  Interview,  audiotape,  and/or  videotape  participants  about  AHLTA  usability 

a.  A  co-PI  and  the  PSE  staff  completed  a  site  visit  to  TAMC  in  May  2007  to 
ensure  site  readiness  for  the  June  data  collection 

1)  Network  connections  were  installed  in  the  designated  office  space 

2)  Power  was  restored  to  a  wall  in  one  of  the  offices 
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3)  All  equipment  was  tested  and  problems  were  solved  (e.g.,  cameras, 
computers) 

4)  Other  equipment  needed  for  June  data  collection  was  acquired 

5)  The  interview  guide  was  developed 

b.  Meet  with  individual  clinicians  -  completed  in  June  2007 

1)  Twelve  individual  clinician  interviews  were  completed 

2)  One  provider  was  ill  and  could  not  be  rescheduled 

3)  Six  naturalistic  observations  were  completed 

4)  During  June  2007,  sufficient  data  were  collected  to  conduct  the  analysis  to 
meet  Aim  1 

c.  Analyze  audio  and  videotapes 

1)  During  June  data  collection,  some  clinician  participants  chose  to  use 
AHLTA  screens  from  their  panels  of  patients  as  they  were  better  able  to 
illustrate  usability  issues.  This  was  not  intended.  The  plan  in  the  protocol 
was  to  have  providers  use  test  patients  to  demonstrate  their  concerns  with 
AHLTA.  The  investigators  knew  these  data  would  need  to  be  de- 
identified.  They  also  brought  this  unintended  occurrence  to  the  attention 
of  the  Human  Use  Committee  (HUC)  at  TAMC.  The  HUC  was  grateful 
for  the  investigators’  forthrightness  and  found  the  plans  to  de-identify  the 
Jun  data  adequate  to  protect  information  from  actual  patients. 

2)  The  analysis  of  the  June  data  was  put  on  hold  due  until  the  patient  data  can 
be  de-identified. 

3)  As  soon  as  documents  are  staffed  through  all  necessary  parties  and 
USAMRAA  gives  the  approval,  PSE  will  de-identify  the  data  and  proceed 
with  the  analysis 

4)  The  work  has  not  progressed  beyond  this  point.  There  are  no  publications, 
to  date,  resulting  from  this  work.  However,  once  the  June  data  are 
analyzed,  a  manuscript  will  be  prepared  to  reflect  the  findings. 

d.  Collate  usability  issues  with  the  current  version  of  AHLTA 

e.  Expand  FDDs 

Task  5:  Develop  the  new  AHLTA  prototypes  with  user-centered  design  principles — 

work  has  not  progressed  to  task  5. 

Task  6:  Complete  pilot  testing — work  has  not  progressed  to  task  6. 

Task  7:  Complete  the  human  performance  study  Aim  2  comparing  the  current 

version  of  AHLTA  and  the  new  AHLTA  prototype — work  has  not  progressed  to  task 

7. 

Task  8:  Data  analysis  and  manuscript  writing — work  has  not  progressed  to  task  8. 

Task  9:  Identify  issues  and  requirements  for  aCPGs 

a)  The  research  team  realized  that  the  actual  aCPG  work  could  begin  sooner  than 

reflected  in  the  linear  task  list. 
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b)  On  2 1  July  2007,  the  PI  and  co-PIs  discussed  general  areas  and  ideas  to  be 
purposed  during  the  aCPG  interviews. 

c)  On  5  August  2007  a  draft  interview  guide  for  collecting  date  related  to  aCPG 
issues  and  requirements  was  shared  with  PSE. 

d)  During  various  team  calls  lively  discussions  took  place  regarding  the  best 
focus  of  the  aCPG  interviews. 

e)  On  3 1  August  2007,  PSE  developed  a  separate  draft  interview  guide. 

Work  has  not  progressed  yet  to  the  remaining  tasks. 

Task  10:  Develop  the  new  aCPGs  in  AHLTA  with  user-centered  design  principles 
Task  11:  Complete  pilot  testing  of  the  aCPG  prototype 

Task  12:  Complete  the  human  performance  study  Aim  4  comparing  the  current 
aCPG  AIM  forms  with  the  new  aCPG  prototype  in  AHLTA 
Task  13:  Data  analysis — 

Task  14:  Completing  the  final  report  and  writing  manuscripts 

KEY  RESEARCH  ACCOMPLISHMENTS: 

•  Completed  4  required  IRB  approvals 

•  Completed  data  collection  about  usability  issues 


REPORTABLE  OUTCOMES: 

•  Developing  a  strong  appreciation  for  conducting  a  research  study  using  an 
electronic  health  record  that  is  still  undergoing  significant  changes 

CONCLUSION: 

•  A  real  challenge  with  this  work,  as  noted,  is  conducting  a  site-specific  research 
study  based  on  the  current  version  of  AHLTA  when  future  releases,  managed, 
developed,  and  deployed  centrally  by  the  MHS,  contain  functionality  that  is 
relevant  to  usability  and  aCPGs.  This  issue  has  been  the  source  of  considerable 
contemplation  by  the  PI  and  coPIs.  Without  attending  to  this  in  a  very  thoughtful 
way,  the  entire  project  could  be  completed  successfully  from  a  research 
standpoint  but  make  very  little  contribution  to  science  or  operations  because  of 
failing  to  answer  the  ‘so  what?’  question. 

•  A  second  significant  challenge  directly  affects  the  ability  to  execute  any/all 
Center  of  Excellence  objectives.  When  the  proposal  was  conceived,  there  was  an 
intention  by  the  Army  Medical  Department  to  work  with  CITPO  to  establish  local 
service  development  environments  to  support  the  ongoing  development  of 
AHLTA.  Due  to  changes  in  leadership,  acquisition  rules  and  regulations,  and 
new  guidance  resulting  from  the  Defense  Business  Transformation  initiative, 
there  have  been  no  MTFs  approved  for  establishing  a  local  service  development 
environment.  This  negatively  impacts  the  ability  of  TAMC  to  serve  as  a  Center 
of  Excellence  for  automating  DoD/VA  approved  CPGs  as  well  as  for 
accomplishing  the  study  objectives. 
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